Over-the-air computation (AirComp) based federated learning (FL) has been a promising technique for distilling artificial intelligence (AI) at the network edge. However, the performance of AirComp-based FL is decided by the device with the lowest channel gain due to the signal alignment property. More importantly, most existing work focuses on a single-cell scenario, where inter-cell interference is ignored. To overcome these shortages, a reconfigurable intelligent surface (RIS)-assisted AirComp-based FL system is proposed for multi-cell networks, where a RIS is used for enhancing the poor user signal caused by channel fading, especially for the device at the cell edge, and reducing inter-cell interference. The convergence of FL in the proposed system is first analyzed and the optimality gap for FL is derived. To minimize the optimality gap, we formulate a joint uplink and downlink optimization problem. The formulated problem is then divided into two separable nonconvex subproblems. Following the successive convex approximation (SCA) method, we first approximate the nonconvex term to a linear form, and then alternately optimize the beamforming vector and phase-shift matrix for each cell. Simulation results demonstrate the advantages of deploying a RIS in multi-cell networks and our proposed system significantly improves the performance of FL.